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Summary 


Background 

The  Navy  Personnel  Research  and  Development  Center  (NPRDC),  working  with  the  Naval 
Sea  Systems  Command  (NAVSEA-04PT)  and  Chief  of  Naval  Education  and  Training  (CNET- 
05),  completed  a  program  of  research,  development,  and  evaluation  for  automating  classroom 
activities  in  which  interactive  electronic  technical  manuals  (lETMs)  are  used.  The  project 
addressed  two  Navy  training  problems:  (1)  the  need  to  utilize  emerging  technologies  to  improve 
maintenance  performance  and  reduce  maintenance  costs  and  (2)  the  need  to  improve  Ae 
efficiency  of  the  Navy  training  pipeline.  These  problems  are  being  addressed  by  combining 
lETM  technology  with  an  integrated  electronic  multimedia  editing  and  delivery  system.  The 
system  takes  advantage  of  advances  in  technology  that  uses  electronic  data,  permits  new  ways  of 
creating  and  delivering  classroom  instruction,  and  provides  standardized  ways  of  interchanging 
various  forms  of  electronic  text  and  graphics. 

Objective 

The  objective  of  this  evaluation  was  to  determine  the  suitability  of  the  lETM/AC  system  for 
developing,  revising,  and  delivering  Navy  instruction  in  the  classroom. 

Approach 

Two  evaluation  areas  are  described.  They  are  (i)  the  technical  aspects  of  the  lETM/AC 
system  and  (2)  the  user  experiences  with  the  lETM/AC  system  reported  during  the  evaluation 
period.  Software  system  development,  specifications,  and  documentation  of  the  technical  system 
criteria  emerged  from  the  development  of  the  system.  System  capabilities  and  characteristics 
were  evaluated  by  conducting  a  technical  overview  of  the  system,  observing  use  of  the  system, 
and  collecting  user  reports  during  pilot  tests.  Questionnaires,  interviews,  and  observation  notes 
were  used  to  record  this  information.  System  hardware  and  software  requirements  were 
documented.  Methods  for  expanding  the  lETM/AC  computer  architecture  to  accommodate  MS- 
Windows  applications  and  IGs  authored  in  a  range  of  word  processing  programs  were  evaluated. 

Information  pertaining  to  user  experiences  during  pilot  usage  of  the  lETM/AC  system  was 
collected.  Pilot  usage  refers  to  instructor,  student,  and  institutional  experiences  with  the 
prototype  system.  These  reported  user  experiences  pertain  to  acceptance  and  satisfaction,  ease- 
of-use,  system  performance,  and  descriptions  of  institutional  characteristics,  effects,  and  site 
activities.  Survey  instruments  were  developed  to  gather  information  from  instructors  and 
students  on  their  use  of  the  system.  Institutional  impacts  were  identified.  Those  impacts  included 
instructional  development  and  delivery,  institutional  efficiency,  overall  acceptance,  and  the 
affect  on  personnel.  Performance  scores  were  obtained  for  a  comparison  of  those  who  attended 
paper-based  versus  automated  courses. 

Condusions 

Overall  the  personnel  interviewed  had  positive  comments  about  the  automated  classroom 
system.  Major  concerns  were  the  need  for  more  experienced  computer  support  personnel,  the 
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perceived  lack  of  quality  graphics  in  the  Gas  Turbine  lETM,  and  a  concern  over  the  reduction  in 
the  length  of  the  training  courses. 

The  lETM/AC  has  tremendous  potential  to  become  tomorrow’s  tradition  by  providing  a  new 
innovative  learning  environment.  Instructors  and  students  learned  to  use  the  lETM/AC  system  in 
a  short  amount  of  time. 

A  comparison  of  student  performance  scores  in  the  traditional  paper-based  classes  and  the 
lETM-based  classes  showed  promising  results.  There  was  slight  improvement  in  overall  average 
student  scores  for  lETM-based  versus  paper-based  courses. 

The  lETM/AC  expanded  system  architecture,  using  Training  Integration  Management 
Software  (TIMS),  accepts  course  materials  from  a  variety  of  sources  including  AIM,  SGh^ 
editors,  instructional  authoring  programs,  and  PC  word  processors.  Augmenting  TIMS  will 
provide  several  benefits  beyond  the  classroom.  Cost  savings  should  be  achieved  by  eliminating 
the  requirement  to  convert  legacy  data  and  course  materials  prepared  with  word  processing  or 
.  other  authoring  programs  to  SGML  format. 

Recommendations 

Choose  the  appropriate  course  for  automation.  The  course  targeted  for  automation  should 
undergo  a  thorough  Instructional  Systems  Design  analysis  before  the  training  materials  are 
developed. 

Conduct  a  site  survey  to  include  required  facilities  modifications,  course  and  personnel 
requirements,  and  visit  an  established  automated  classroom  to  learn  from  other’s  experiences. 

Keep  IGs  and  TGs  separate  from  the  lETM.  Course  materials  should  not  be  embedded  in  the 
lETM.  This  will  save  money  and  time  when  either  the  course  or  the  lETM  requires  revision; 
there  will  be  no  need  to  return  to  the  lETM  source  for  curriculum  revisions. 

Billets  should  be  established  for  systems  administration  and  life  cycle  maintenance  of 
hardware,  software,  and  courses.  Existing  instructors  and  support  personnel  cannot  provide  the 
necessary  support  without  additional  training  and  billets. 

Learning  Resource  Centers  should  be  integrated  with  the  classrooms,  be  staffed  by 
instructors  and  be  available  for  after  hours  study.  This  will  free  up  classroom  space  for  other 
courses  and  provide  needed  student  access  for  after  hours  studying. 

Multiple  classrooms  should  be  networked  with  a  central  file  server.  This  will  allow  sharing 
of  data,  easy  reconfiguration  of  classrooms  with  different  courses,  and  make  course  maintenance 
more  efficient.  Life  cycle  maintenance  and  configuration  management  are  more  difficult  if 
classrooms  are  not  networked. 

Instructor  workstations  should  be  available  in  the  instructor  preparation  areas.  This  will 
allow  them  to  personalize  their  materials  and  prepare  for  lectures  without  tying  up  the 
classroom. 
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Introduction 


Background 

The  Navy  Personnel  Research  and  Development  Center  (NPRDC),  working  with  the  Naval 
Sea  Systems  Command  (NAVSEA-04PT)  and  Chief  of  Naval  Education  and  Training  (CNET- 
05),  completed  a  program  of  research,  development,  and  evaluation  for  automating  classroom 
activities  in  which  interactive  electronic  technical  manuals  (lETMs)  are  used.  The  project 
addressed  two  Navy  training  problems:  (1)  the  need  to  utilize  emerging  technologies  to  improve 
maintenance  performance  and  reduce  maintenance  costs  and  (2)  the  need  to  improve  the 
efficiency  of  the  Navy  training  pipeline.  These  problems  are  being  addressed  by  combining 
lETM  technology  with  an  integrated  electronic  multimedia  editing  and  delivery  system.  The 
system  takftg  advantage  of  advances  in  technology  that  uses  electronic  data,  permits  new  ways  of 
creating  and  delivering  classroom  instruction,  and  provides  standardized  ways  of  interchanging 
various  forms  of  electronic  text  and  graphics. 

In  1993,  NAVSEA  tasked  Martin  Marietta  to  design  a  system  to  deploy  lETMs  in  Navy 
classrooms.  Their  objective  was  to  produce  an  integrated  database  which  would  be  used  aboard 
'ship  for  improved  maintenance  performance  and  in  the  schoolhouse  to  provide  a  dynamic, 
interactive  training  environment,  thus  improving  shipboard  performance,  enhancing  training, 
and  reducing  course  length.  The  Gas  Turbine  Technician  Journeyman  Training  or  “C”  school  at 
Great  Lakes  Training  Center,  IL  was  chosen  as  the  test  bed. 

NAVSEA  focused  on  the  technical  applications-  of  lETMs  in  the  classroom  and  defined  the 
scope  of  the  lETM  database,  developing  the  maintenance  diagnostic  and  operations 
viewpackages,  and  electronically  enhanced  Instructor  Guides  (IGs)  and  Trainee  Guides  (TGs). 

NPRDC  evaluated  the  impact  of  the  automated  classroom  on  student  performance,  course 
content  and  length,  as  well  as  time  and  cost  issues.  NPRDC  and  NAVSEA  developed 
implementation  recommendations  and  system  specifications.  The  lETM/AC  architecture  was 
expanded  to  accept  curriculum  materials  developed  with  a  variety  of  word  processing  and 
curriculum  authoring  programs. 

Products  resulting  from  this  effort  include  (1)  an  instructional  delivery  architecture  using 
format  neutral  components,  known  as  the  Training  Integration  Management  Software  (TIMS), 
(2)  an  evaluation  that  documents  the  effects  of  an  automated  instructional  system,  (3) 
enhancement  of  the  automated  GS  Electrical  (course  identification  number  A-652-0328)  and  GS 
Mechanical  Maintenance  (course  identification  number  A-652-0303)  courses,  (4)  a  cost/benefits 
analysis  report,  (5)  a  survey  of  lETMs  used  for  training  and  education  report,  and  (6) 
implementation  recommendations  and  system  specifications. 

The  Advanced  Development  sponsor  for  this  effort  was  LCDR  Tim  Steele  (PERS-OOHl  1). 
The  Implementation  sponsor  was  Ms.  Saundra  Drummer  (CNET  N5214). 
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Objective 

The  objective  of  this  evaluation  was  to  detennine  the  suitability  of  the  lETM/AC  system  for 
developing,  revising,  and  delivering  Navy  instruction  in  the  classroom. 

Approach 

Two  evaluation  areas  are  described.  They  are  (1)  the  technical  aspects  of  the  lETM/AC 
system  and  (2)  the  user  experiences  with  the  lETM/AC  system  reported  during  the  evaluation 

period. 

Technical  Aspects  of  the  System 

Software  system  development,  specifications,  and  documentation  of  the  technical  system 
criteria  emerged  from  the  development  of  the  system.  This  documentation  describes  the  system 
and  subsystem  components  and  software  capabilities.  System  capabilities  and  characteristics 
were  evaluated  by  conducting  a  technical  overview  of  the  system,  observing  use  of  the  system, 
and  collecting  user  reports  during  pilot  tests.  Questionnaires,  interviews,  and  observation  notes 
were  used  to  record  this  information.  System,  hardware  and  software  requirements  were 
documented.  Methods  for  expanding  the  lETM/AC  computer  architecture  to  accommodate  MS- 
Windows  applications  ^d  IGs  authored  in  a  range  of  word  processing  programs  were  evaluated. 

User  Experiences 

Pilot  usage  refers  to  instructor,  student,  and  institutional  experiences  with  the  prototype 
system.  These  pertain  to  acceptance  and  satisfaction,  ease-of-use,  system  j^rformance,  and 
descriptions  of  institutional  characteristics,  effects,  and  site  activities.  Survey  instruments  were 
developed  to  obtain  information  from  instructors  and  students  on  their  use  of  the  system. 

Instructors  were  asked  to  estimate  the  time  and  effort  to  translate  information  into  the 
electronic  system,  to  update  instructional  materials  when  revisions  were  necessary,  to  prepare  for 
their  lectures  using  the  system,  and  to  be  trained  on  the  system.  Instructors  also  were  asked  to 
evaluate  the  use  of  this  system  for  delivering  instructional  materials,  overall  classroom 
environment,  and  to  provide  suggestions  for  improving  the  system. 

Students  were  asked  to  evaluate  the  system  for  ease-of-use  and  the  utility  of  the  student 
workstation.  Instructor  comments  on  performance  while  using  the  system,  overall  classroom 
environment,  impact  on  student  learning,  and  suggestions  for  system  improvement  were 
obtained.  Performance  scores  were  obtained  for  students  attending  paper-based  and  automated 
courses. 

Instimtional  impacts  were  identified.  Those  impacts  included  instructional  development  and 
delivery,  instimtional  efficiency,  overall  acceptance,  and  the  affect  on  personnel.  Costs  and 
impacts  associated  with  implementation  of  the  lETM/AC  system  were  identified  and  discussed 
in  a  cost  benerits  analysis  report  (Mark,  Kribs,  Morris,  &  Dickason,  1996). 
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lETM/AC  System 

The  classroom  is  designed  to  use  Commercial  Off  The  Shelf  (COTS)  hardware  and  software. 
The  Gas  Turbine  lETM  utilizes  the  Mo  Access  GUIDE™  product  for  computer  display  of  the 
technical  information.  Over  110,000  pages  of  technical  information  were  converted  to  electronic 
format  and  Standard  Generalized  Markup  Language  (SGML)  tagged  during  the  development  of 
the  GS  lETM.  Converted  data  were  uploaded  into  an  object  oriented  database  using  relational 
database  management  software.  The  GSE  and  GSM  course  materials  were  embedded  in  the 
BETM.  To  create  lETM  functionality,  viewpackages  for  general  knowledge,  troubleshooMg, 
maintenance,  and  operations  were  developed.  Linking  the  lETM  to  the  IG  and  other  applications 
was  requited  for  computer  presentation  of  the  electronic  “book”  with  hypertext/hyperlink 
capability.  This  lETM  development  process  was  examined  and  a  survey  of  lETMs  used  in 
education  anH  training,  both  in  other  services  and  in  the  civilian  community,  was  conducted 
(Kiibs,  Mark,  Morris,  &  Dickason,  1996). 

The  Mine  Countermeasures  (MCM)  Isotta  Frachini  lETM,  piloted  with  the  Training 
Integration  Management  Software  (TIMS),  is  displayed  by  the  Electronic  Book  DynaText™ 
product.  The  MCM  course  materials  were  authored  in  Word  Perfect  and  not  SGML  tagged. 
Other  automated  classrooms  may  require  different  hardware  and  software,  depending  on  the 
course,  the  lETM,  and  facility  requirements.  Figure  1  provides  a  graphic  of  the  present 
automated  classroom  configuration. 

Instructor  Station  Hardware  Configuration 

This  instructor  station  configuration  included  the  following  COTS  software:  MS  DOS  6.22, 
Windows  for  Work  Groups  3.11,  MS  ACCESS,  Guide  Reader,  Video  for  Windows,  and  SYTOS 
tape  back-up  package. 

•  486  DX/4 100  MHz  CPU 

•  16  megabyte  RAM 

•  3.5”  1.44  megabyte  floppy  drive 

•  6  gigabyte  hard  drive 

•  3  X  CD-ROM  drive 

•  32  bit  Sound  Blaster  card 

•  32  bit  LAN  card 

•  Tape  Drive 

•  20”  SVGA  monitor 

•  Keyboard 

•  Mouse 

•  Laser  Printer 

•  VCR 
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Student  Station  Hardware  Configuration 

The  student  station  configuration,  at  the  time  of  the  study,  included  the  following  COTS 
software:  MS  1X)S  6.22,  Windows  for  Work  Groups  3.11,  MS  ACCESS  ,  Guide  Reader,  and 
Video  for  Windows. 

•  486  DX/4  100  MHz  CPU 

•  16  megabyte  RAM 

•  3.5”  1.44  megabyte  floppy  drive 

•  2.572  gigabyte  hard  drive 

•  3  X  CD-ROM  drive 

•  32  bit  Sound  Blaster  card 

•  32  bit  LAN  card 

•  17”  SVGA  monitor 

•  Keyboard 

•  Mouse 

Automated  Classroom  Components 

•  EtherNet  LAN  to  connect  instructor  station  (server)  to  student  stations  (clients) 

•  Xerox  liveBoard,  67”  projection  monitor  with  VGA  resolution  of  640x480,  with  an  Intel 
486  DX/2  66  MHz  CPU 

•  LINK  Video  Networking  System  to  allow  instructor  to  view  student  monitor  displays  on 
his  screen,  switch  and  display  to  Xerox  LiveBoard  or  display  on  all  monitors. 


4 


Figure  1.  Autmnated  classroom  configuration 


Subjects 

Forty-eight  enlisted  military  students,  attending  the  lETM  gas  turbine  “C”  school  courses 
participated  in  the  study.  The  majority  of  these  students  were  rated  E-5  and  below.  In  generd 
they  had  returned  from  the  fleet,  rather  than  directly  from  a  prior  school,  to  attend  these  “C” 
school  courses. 

Nine  enlisted  military  instructors  participated  in  the  study.  These  Instructors  were  rated  E-6 
and  E-7. 


Data  Collection  Methods  and  Instruments 

Information  was  obtained  by  observing  the  classroom  in  use,  conducting  structured 
interviews  with  instructors,  students,  and  support  personnel,  surveying  instructors  and  students, 
and  reviewing  test  scores  from  previous  (paper-based)  as  well  as  the  present  automated  classes. 
Test  scores  for  the  Gas  Turbine  courses,  GSE  (A-652-328)  and  GSM  (A-652-303),  were 
obtained  from  the  Instructional  Support  System  (ISS)  database. 


Survey 

A  survey  was  designed  to  gather  experiences  using  the  system,  both  from  students  and 
instructors.  Survey  statements  were  rated  on  a  five-point  scale,  “Strongly  Agree  to  Strongly 
Disagree,”  and  were  followed  by  open-ended  questions.  The  survey  was  administered  to  students 
and  instructors  during  instruction. 

Test  scores 

Test  scores  were  obtained  for  written  tests  and  final  course  average.  The  final  course  average 
consisted  of  a  final  exam  (40%),  laboratory  exercises  (40%),  homework  (10%),  and  quizzes 
(10%).  The  tests  were  paper-based. 

Training  Integration  Management  Software  (TIMS) 

The  lETM/AC  computer  architecture  was  expanded  to  use  MS-Windows  standards  and  will 
accommodate  lesson  plans  and  Instructor  Guides  (IGs)  from  a  variety  of  sources  including  the 
Authoring  Instructional  Materials  (AIM)  system,  SGML  editors,  and  PC  word  processors.  This 
expanded  architecture,  which  can  use  a  range  of  inputs,  is  referred  to  as  “format  neutral. 
Training  Integration  Management  Software  (TIMS),  a  software  module  that  manages  a  database 
of  annotations  for  IGs  and  TGs,  supports  format-neutral  processing.  TIMS  was  piloted  from  July 
through  September  1995  and  is  still  being  used  for  the  MCM  course.  This  course  provided  a 
testbed  for  the  development  of  a  TIMS  Word  Perfect  browser  which  is  described  below. 

Figure  2  below  shows  the  Main  Screen  (following  a  login  procedure)  of  the  TIMS  package. 
The  Main  Screen  is  where  all  tasks  are  performed  using  TIMS.  It  actually  consists  of  four 
separate  panes.  The  Control  pane  is  the  narrow  vertical  window  along  the  left  side  of  the  display 
in  Figure  2.  This  pane  contains  buttons  which  control  the  other  three  panes.  The  Instructor  Guide 
(IG)  pane  always  occupies  the  top  part  of  the  display.  This  is  where  the  IG  is  displayed.  The 
Notes  pane  is  located  below  the  left  side  of  the  IG  pane.  This  pane  contains  any  personal 
annotations  that  has  been  added  by  the  instructor.  The  Related  Instructor  Activity  (RIA)  pane 
lies  to  the  right  of  the  Notes  pane.  This  pane  contains  any  annotations  that  have  been  added  by 
the  curriculum  developer. 
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Figure  2.  TIMS  screen  layout 

The  TIMS  software  components  consist  of  commercial-off-the-shelf  (COTS)  MS-Windows 
compatible  programs.  The  software  components  include: 

Browsers.  Browsers  support  the  viewing  of  information  located  in  curriculum  databases. 
This  information  can  be  in  many  formats.  For  example,  browsers  allow  instractors  to  display  IGs 
created  in  Word  Perfect  or  AIM.  Instructors  cannot  edit  the  actual  IG,  but  can  add  personalized 
notes  known  as  annotations.  Annotations  can  be  linked  to  topics  discussed  in  the  IG  for 
classroom  presentation.  A  browser  supports  loading,  adding,  deleting,  and  saving  annotations. 
The  annotations  may  be  text,  animations,  graphics,  or  multimedia.  Instructors  have  the  capability 
to  page  up  and  down  within  the  IG,  move  from  paragraph  to  paragraph,  and  zoom  in  and  out.  IG 
navigation  features  are  accessible  from  a  remote  control  unit  as  well  as  keyboard  and  mouse.  A 
Word  Perfect  browser  is  presently  in  use  and  Guide  and  AIM  browsers  are  being  developed. 

Instructor’s  Annotation  Editor.  This  program  allows  instructors  to  directly  enter  text 
annotations  and  supports  the  editing  of  other  media  such  as  animations.  When  an  annotated 
paragraph  in  the  IG  is  selected,  the  linked  annotations  are  displayed.  These  annotations  can  be 
directed  to  and  displayed  on  student  workstations  or  to  a  large  screen  such  as  an  automated 
LiveBoard.  Editor  features  are  accessible  from  a  remote  control  unit  as  well  as  keyboard  and 
mouse. 

Student’s  Annotation  Viewer.  This  program  allows  the  instructor’s  annotations  to  be 
viewed  on  student  workstations.  Instructors  have  control  over  what  annotations  are  displayed. 

Network  Protocol.  This  software  allows  instructor  and  student  workstations  to  exchange 
data. 
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File  Sharing  between  Instructor/Student  Workstations.  This  capability  allows  student 
workstations  to  share  annotation  data  with  the  instructor’s  worlutation.  An  example  is  a  graphic 
file  that  the  instructor  can  broadcast  to  the  student  workstation  during  lecture.  File  sharing 
allows  a  student  to  review  the  graphic  or  text  while  completing  homework  assignments.  Anomer 
example  is  an  animation  file  that  could  aid  in  understanding  a  difficult  concept  during  after 

hours  study. 

Remote  Control  Unit  for  Instructor.  An  infared  remote  control  unit  allows  instructors  to 
navigate  through  the  IG  and  its  annotations  from  anywhere  in  the  classroom.  This  is  an 
advantage  over  the  usual  method  where  the  instructor  is  required  to  remain  close  to  the  computer 
and  podium  as  he  lectures. 

Instructor’s  Response  Module.  This  program  allows  an  instructor  to  activate  a  response 
pad  on  the  individual  student  workstations  for  conducting  classroom  quizzes.  Student  responses 
are  stored  and  displayed  on  the  instructor  workstation.  The  response  pad  accommrri^es 
True/False,  Yes/No,  and  multiple  choice  responses.  The  responses  are  displayed  on  a  desk-by¬ 
desk  basis,  and  as  a  bar  graph  for  the  entire  class.  The  response  displays  are  color  keyed  for  easy 

viewing  by  the  instructor. 

Student’s  Response  Module.  This  program  displays  a  response  pad  on  the  student 
workstation  as  directed  by  the  instructor  workstation.  The  response  pad  provides  for  the  same 
responses  as  described  above  for  the  Instructor  s  response  module. 

User  Interface.  An  effective  graphic  user  interface  has  been  implement^  to  ensure  that 
information  is  presented  clearly,  the  system  is  easy  to  use,  and  there  is  an  artistic  sense  of  layout 
and  balance.  Attention  has  been  focused  on  functionality  and  placement  of  “buttons  or  icons 
that  represent  links  to  annotations  and  navigation  capabilities. 

Student  Performance 


A  comparison  of  student  performance  scores  in  the  traditional  paper-based  classes  and  the 
lETM-based  classes  showed  promising  results  as  documented  in  the  costs/benefits  report  (Mark, 
Kribs,  Morris,  &  Dickason,  1996).  The  test  scores  for  six  paper-based  GSE  and  GSM  classes 
were  obtained  and  used  as  a  baseline.  Test  scores  for  seven  GSE  ETM-based  classes  and  three 
GSM  lETM-based  classes  were  available  for  comparison  at  the  time  the  report  was  prepwM. 
There  was  slight  improvement  (Figures  3  &  4)  in  overall  average  student  scores  of  91.88  for 
GSE  lETM-based  versus  88.66  GSE  paper-based  and  92.23  for  GSM  lETM-based  versus  90.60 
for  GSM  paper-based  courses. 
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Instructor  Survey 

Nine  instructors  completed  a  written  survey  of  their  experiences  while  teaching  from  the 
lETM/AC  system.  Half  of  the  instructors  had  two  or  more  years  teaching  experience.  Overall, 
they  ranged  from  less  than  one  year  to  a  maximum  of  10  years  teaching  experience.  Six 
instructors  had  previous,  but  limited,  computer  experience  and  none  of  them  had  taught  ivith 
computers.  The  survey  gathered  demographic  information,  and  experiences  related  to  system 
functionality,  classroom  interaction,  equipment  used,  subject  niatter,  and  facilities.  A  five  point 
scale  was  used  for  the  responses.  Strongly  agree  had  a  value  of  five  and  strongly  disagree  had  a 
value  of  one.  The  instructors’  responses  are  shown  in  Figure  5  with  the  mean  response  value  for 
each  item  displayed  in  the  far  right  column. 

Instructors  reported  the  use  of  the  computer  made  the  course  more  interesting  and  supported 
the  use  of  new  media  in  the  classroom,  but  were  undecided  or  doubtful  about  how  the  system 
was  affecting  the  flow  of  information.  More  software  than  hardware  problems  were  reported. 
There  were  mixed  feelings  about  the  amount  of  class  time  lost  due  to  program  failures.  In  spite 
of  some  problems,  instructors  felt  the  computer  should  be  used  to  enhance  training  and  that  the 
system  will  result  in  less  time  required  to  teach  a  course. 

Only  one  instructor  agreed  with  the  statement,  “I  prefer  the  use  of  this  system  when 
compared  to  paper-materials”.  This  seems  surprising  given  a  majority  of  instructors  agreed  Aat 
the  computer  should  be  used  in  other  classes  and  the  computer  enhanced  student  learning.  One 
explanation  is  that  they  felt  this  particular  system  did  not  meet  their  requirements  for  teachmg 
even  though  they  think  that,  in  general,  computers  are  a  positive  addition  to  the  classroom. 

At  the  end  of  the  survey,  instructors  were  given  space  to  provide  suggestions  or  comments 
about  the  system.  Instructors  felt  information  was  easier  to  locate  using  the  system.  They  liked 
the  control  the  system  gave  them  over  what  students  saw  on  their  workstations,  the  use  of  stete- 
of-the-art  equipment  to  display  the  training  materials  and  drawings,  and  stated  that  the  facilities 
provided  a  nice  comfortable  learning  environinent. 

Instructors  were  concerned  about  the  students  entering  their  notes  on,  the  computer  during 
class.  Students  lacked  typing  skills,  which  made  note  taking  on  the  computer  time-consuming 
and  distracting  during  lecture.  Instructors  suggested  improvements  could  be  made  to  the 
UveBoard  resolution  and  pens.  The  graphics  were  reported  to  be  of  poor  quality,  probably 
because  they  were  scanned  from  technical  manuals  rather  than  re-authored  using  graphics 

software. 
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Use  of  the  com  puter  did  not  adversely 
affect  inform  ation  flow  In  the  course. 

Use  of  the  com  puter  made 
the  course  more  interesting 

I  like  to  see  new  m  edia  in 
the  classroom 


I  felt  confident  w  hen  using  the 
com  puter. 

There  were  few  problems  with 
the  hardware 

There  were  few  problems 
w Ith  the  softw are. 

Little  class  time  was  lost 
because  of  program  failures 

The  com  puter  should  be  used  in 
other  courses . 


Use  of  the  computer  enhanced 
student  learning. 


Training  on  using  the  com  puter  was 
adequate. 

Training  on  using  the  software  was 
adequ^e. 

I  wiil  be  able  to  cover  the  course 
m  aterial  in  less  tim  e  using 

The  use  of  this  system  will  be  useful 
forthe  LCM  of  the  course 


I  prefer  the  use  of  this  system  when 
com  pared  to  a  paper-based  . 

01  23  4  567  89 


B  AGREE 

□  UNDECIDED 

□  DISAG  REE 


*  Scale  Average 


N  urn  ber  of  Respondents 


Figure  5.  Instructor  survey  results 


Structured  Interviews 

Structured  interviews  were  conducted  with  supervisors,  managers,  and  support  personnel. 
Instructors  were  interviewed  in  pairs  from  both  the  GSE  and  GSM  content  areas. 

Instructors 

A  total  of  eight  instructors  were  interviewed,  six  GSE  and  two  GSM  instructors.  In  general 
instructors  thought  the  automated  classroom  and  the  lETM  had  a  lot  of  potential  for  improving 
the  current  courses  of  instruction.  However,  they  were  greatly  concerned  about  the  “final 
product”  (the  trainees  being  sent  to  the  Fleet)  in  the  current  implementation.  These  concerns 
centered  around  three  main  areas:  (1)  The  lETM,  (2)  the  reduction  in  course  length,  and  (3)  the 
need  to  better  understand  how  to  teach  in  the  new  electronic  environment.  The  concerns  about 
the  lETM  centered  around  the  poor  quality  of  the  lETM  graphics.  This  was  especially  evident 
with  the  GSE  instructors  who  had  to  use  the  wiring  schematics  for  instruction.  A  majority  of 
instructors  voiced  concerns  about  the  reduction  in  course  length.  One  of  eight  instructors  was 
concerned  about  the  revisions  made  to  the  courses.  Several  instructors  stated  that  they  felt  the 
graduates  of  the  automated  classrooms  were  not  as  well  trained  as  previous  graduates.  They 
thought  the  increase  in  performance  scores  that  the  lETM  classes  had  shown  were  due  to  easier, 
multiple  choice  tests  and  smaller  class  sizes.  The  third  area  of  concern  was  their  need  to 
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understand  how  to  teach  in  the  new  classrooms.  While  the  majority  felt  comfortable  with  the 
mechanics  of  using  the  new  technology  they  were  less  comfortable  with  the  most  effective  way 
to  use  the  technology  to  “get  the  point  across.” 

Supervisory  personnel 

Two  of  the  supervisory  personnel  responsible  for  the  GSE  and  GSM  areas  were  interviewed. 
Their  primary  concem  was  centered  on  the  amount  of  effort  required  to  get  the  automated 
classrooms  up  and  running  and  to  keep  them  running.  Since  computer  technology  is  evolving  so 
quickly  there  is  a  felt  need  to  plan  for  equipment  upgrades  on  an  almost  continual  basis.  Related 
to  that  is  the  need  for  additional  billets  to  support  the  automated  classrooms.  Currently, 
personnel  who  have  some  computer  expertise  are  assigned  computer  support  tasks.  have  no 
special  training  and  are  basically  learning  on  the  job.  This  assignment  may  be  in  addition  to  their 
regular  duties  in  the  schoolhouse. 

Support  personnel 

Three  personnel  who  man  the  DETM  support  office  were  interviewed.  They  have 
responsibilities  that  have  accumulated  over  time.  Their  concerns  were  centered  on  the  need  for 
additional  billets  for  classroom  support  as  well  as  the  need  to  provide  additional  levels  of 
security  for  the  existing  computer  systems.  They  would  like  to  see  a  system  that  allows 
instructors  and  trainees  access  to  only  that  part  of  the  system  they  need  to  fulfill  their 
responsibilities. 

Overall  the  personnel  interviewed  seemed  to  have  positive  comments  about  the  automated 
classrooms.  Major  concerns  were  the  need  for  more  experienced  computer  support  personnel, 
the  perceived  lack  of  quality  graphics  in  the  Gas  Turbine  lETM,  and  a  concem  over  the 
reduction  in  the  length  of  the  training  courses. 

Student  Survey 

Forty-eight  students,  26  GSE  and  22  GSM,  completed  the  survey.  We  found  that  60%  of  the 
students  bad  previous  computer  experience,  mostly  two  years  or  less,  but  few  had  used 
computers  in  a  classroom  setting.  Keyboarding  or  typing  skills  varied  greatly,  with  most  in  the 
hunt-peck  or  novice  categories. 

Figure  6  summarizes  the  responses  made  in  the  survey.  The  survey  gathered  demographic 
information,  and  experiences  related  to  system  functionality,  classroom  interaction,  equipment 
used,  subject  matter,  and  faciUties.  A  five  point  scale  was  us^  for  coding  responses,  with 
strongly  agree  having  a  value  of  five  and  strongly  disagree  having  a  value  of  one.  The  mean 
response  value  for  each  statement  is  displayed  in  the  far  right  column  of  Figure  6. 

Students  expressed  a  mixed  reaction  to  the  computer  and  how  smoothly  the  information 
flowed  during  lectures.  They  liked  the  capability  to  access  the  lETM  quickly  during  tests, 
supported  the  use  of  new  media  in  the  classroom  and  thought  the  computers  made  the  course 
more  interesting.  Most  of  the  students  felt  the  instructor  was  confident  using  the  system.  As  for 
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hardware  and  software  problems,  opinions  were  varied.  Students  knew  this  was  a  new  system, 
were  aware  of  equipment  failures,  and  recognized  that  the  software  needed  refinement. 

The  next  section  of  the  survey  asked  students  to  respond  to  open-ended  questions.  When 
aclfffH  for  suggestions  to  improve  use  of  this  technology  in  the  classroom,  students  were  mostly 
concerned  with  improving  the  graphics.  Other  suggestions  included  completing  the  program 
(some  hyperlinks  were  not  linked  to  text  or  graphics  in  the  prototype),  improving  monitor 
resolution,  and  improving  the  system  performance.  Four  students  reported  there  was  a  need  to 
use  paper  schematics  and  diagrams.  Seven  students  wanted  to  see  keyboarding  and  computer 
training  offered  before  the  course. 

The  last  section  of  the  survey  provided  space  for  students  to  provide  suggestions  or 
comments  about  the  classroom  system.  They  were  given  these  topics  to  consider:  classroom 
interaction,  instructional  strategies,  equipment  used,  subject  matter/curriculum,  and  facilities. 
Students  liked  using  the  computer  and  having  the  information  displayed  on  the  large 
liveBoard''^  in  the  front  of  the  classroom.  Most  comments  focused  on  the  state-of-the-art 
system,  which  provided  easy  access  to  information  and  diagrams.  Students  liked  the  animated 
drawings  and  found  the  training  materials  useful  and  easy  to  reference.  They  found  it  more 
difficult  to  trace  signals  on  the  LiveBoard  and  suggested  paper  might  be  more  suitable  for  this 
application. 


Use  of  the  com  puter  did  not  adversely 
affect  Information  flow  in  the  course. 

Use  of  the  com  puter  m  ade  the 
course  more  Interesting 

I  like  to  see  new  media  in 
the  classroom 

The  Instructor  seemed  confidant  when 
using  the  computer. 

There  wore  few  problems  with 
the  software 

There  were  few  problems 
with  the  hardware 
Little  class  time  was  lost 
because  of  program  failures 

The  computer  should  be  used . 

Teaching  from  the  electronic  IG  did  not 
adversely  affect  my  learning . 


*  Scale  Average 


Number  of  Respondents 


Figure  6.  Student  survey  results 


Institutional  Impacts 

Interviews  with  school  administrators  and  instructors  revealed  many  institutional  impacts  as 
a  result  of  implementing  the  BETM/AC  system.  The  first  few  Gas  Turbine  instructors  received 
two  weeks  of  lETM/AC  system  training  from  the  contractor.  These  instructors  in  turn  trained 
other  instructors  to  use  the  system.  This  put  additional  time  constraints  on  their  already  busy 
schedules.  Corporate  knowledge  of  this  type  is  very  perishable  and  places  a  burden  on  the  school 
as  the  military  instructors  transfer  to  other  duty  stations. 

As  stated  in  a  lessons  learned  document  (Fretz,  1995),  the  necessity  for  quickly  developing 
the  lETM/AC  system  led  to  some  software  selections  that  proved  inadequate.  Windows  for 
Workgroups  provided  a  quick  and  simple  networking  capability  at  the  time,  but  it  lacks  the  more 
robust  network  administration  and  security  features  needed  for  long  term  use  in  the  schoolhouse. 
The  contractor  combined  the  IG/TG  with  the  lETM,  which  later  proved  to  be  problematic.  When 
a  course  or  lETM  has  been  revised,  the  contractor  must  make  the  changes  and  produce  a  new 
CD-ROM.  The  course  should  not  be  sent  to  the  fleet  as  part  of  the  lETM  CD-ROM.  The  training 
organization  loses  control  over  the  instructional  materials  meant  for  classroom  training. 

Along  with  the  lETM/AC  system  comes  a  requirement  for  additional  support  staff  for  system 
administration,  maintenance,  security,  and  life  cycle  maintenance  of  the  courses.  Billets  have  not 
been  established  to  meet  these  demands.  The  existing  staff  has  taken  on  these  responsibilities  as 
collateral  duties,  but  this  is  not  a  long  term  solution.  Personnel  have  been  assigned  to  these  tasks 
with  a  minimum  of  training  and  experience.  As  for  course  development  and  maintenance, 
responsibilities  need  to  be  defined  for  the  curriculum  developers  and  instructors.  For  example, 
when  a  course  or  lETM  changes  someone  must  establish  software  links  between  the  IG  and  the 
lETM  and  provide  the  instructor  with  the  latest  version. 

The  size  of  the  classrooms  limited  the  throughput  by  lowering  the  class  size  from  a 
mflTcimnm  of  25  Students  to  12  or  15.  Instructors  and  students  had  to  adapt  to  this  change  in  their 
classroom  environment.  Instructors  are  learning  new  presentation  techniques  and  becoming 
innovative  as  they  leam  new  methods  for  annotating  their  IGs.  On  the  student  side,  there  are  no 
instructional  materials  to  study  from  after  hours.  Computers  are  not  available  in  the  barracks. 
Therefore,  students  must  return  to  the  classroom  to  study  and  complete  assignments.  This 
requires  a  better  long  term  solution,  perhaps  integrating  the  classrooms  with  the  Learning 
Resource  Center. 


Conclusions 

Early  efforts  to  develop  automated  classrooms  were  problematic  due  to  immature  software 
for  developing  and  displaying  graphics  and  text  information  (Kribs,  1994).  Because  of 
increasing  instructional  development  and  maintenance  costs,  making  use  of  advancing  computer 
technologies  offers  the  most  efficient  alternative.  In  the  past,  there  was  considerable  time  and 
cost  associated  with  developing  automated  systems  for  classroom  training.  Initially,  computers 
powerful  enough  for  these  applications  were  much  more  expensive  and  there  were  few  programs 
available  that  supported  development  of  materials  that  were  SGML  compatible.  There  were 
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many  risks  associated  with  system  development,  the  need  for  custom  software,  course 
development,  and  networking  hardware  and  software. 

Advances  in  technology  based  on  electronic  data  conversion,  storage,  transfer  and  standards 
have  provided  the  basis  for  developing  prototype  instructor  and  student  workstations  that  may 
provide  significant  advances  in  instructional  development  and  presentation  effectiveness,  and 
efficiencies  in  costs  (Kribs,  Mark,  Morris,  &  Dickason,  1996;  Mark,  Kribs,  Morris,  &  Dickason, 
1996).  It  is  now  possible  to  include:  (a)  high  quality  digital  multimedia,  (b) 
hypertext/hypermedia,  (c)  the  capability  to  store  digital  media  materials  more  efficiently,  and  (d) 
transmission  of  multimedia  through  local,  wide  area,  and  worldwide  networks.  There  are, 
however,  very  few  instances  of  this  technology  being  fully  integrated  into  a  Navy  training 
enviroiunent. 

An  important  aspect  of  the  current  lETM/AC  system  is  the  use  of  COTS  hardware  and 
software,  and  that  the  classroom  is  not  dependent  on  any  one  feature  or  vendor.  As  new 
computer  and  instructional  technology  become  available  the  development  and  presentation 
packages  can  be  utilized  in  the  existing  classroom. 

The  lETM/AC  expanded  system  architecture,  using  TIMS,  accepts  course  materials  from  a 
variety  of  sources  including  AIM,  SGML  editors,  instructional  authoring  programs,  and  PC 
word  processors.  Augmenting  TIMS  will  provide  several  benefits  beyond  the  classroom.  Cost, 
savings  should  be  achieved  by  eliminating  the  requirement  to  convert  legacy  data  and  course 
materials  prepared  with  word  processing  or  other  authoring  programs  to  SGML  format.  Typical 
costs  for  translating  materials  from  standard  word  processing  formats  to  SGML  are  several 
dollars  a  page,  and  millions  of  pages  of  such  materials  are  presently  used  in  Navy  classrooms. 

The  prototype  lETM/AC  contains  instructor  and  student  workstations  that  support  instructor- 
student  interactions,  presentation  of  the  curriculum,  student  practice  and  testing,  and  monitoring 
of  student  responses.  These  functions  are  accomplished  with  automated  IGs  which  guide 
instructor  activities,  local  area  networks  for  interconnecting  students  and  instructors,  student 
response  monitoring,  and  instructor  controlled  multimedia  presentations. 

The  lETM/AC  has  tremendous  potential  to  become  tomorrow’s  tradition  by  providing  a  new 
innovative  learning  environment.  Instructors  and  students  learned  to  use  the  BETM/AC  system  in 
a  short  amount  of  time.  With  the  point  and  click  Windows  interface,  rapid  information  retrieval 
is  possible.  Students  quickly  became  familiar  with  the  functionality  of  the  system.  In  just  a  few 
days,  students  became  comfortable  using  the  computer  for  learning.  In  the  future,  as  students  and 
instructors  have  more  computer  experience,  this  transition  will  become  easier.  Instructors  will 
become  more  innovative  as  they  realize  the  potential  of  the  DETM/AC. 

Reduction  in  training  time  and  course  length  have  resulted  in  cost  savings.  Additional 
savings  are  projected  due  to  reduced  time  and  effort  for  course  maintenance  and  reproduction  of 
materials  (Mark,  Kribs,  Morris,  &.  Dickason,  1996).  As  should  be  done  before  any  course  is 
automated,  the  GSE  and  GSM  courses  were  revised.  The  resultant  efficiencies  are  due  to  a 
thorough  course  review,  rapid  retrieval  of  information  in  the  classroom,  and  less  instructor 
lecture  preparation  time. 
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Recommendations 

Choose  the  appropriate  course  for  automation.  The  course  targeted  for  automation  should 
undergo  a  thorough  Instructional  Systems  Design  analysis  before  the  training  materials  are 
developed. 

Conduct  a  site  survey  to  include  required  facilities  modifications,  course  and  personnel 
requirements,  and  visit  an  established  automated  classroom  to  learn  from  other  s  experiences. 

Keep  IGs  and  TGs  separate  from  the  lETM.  Course  materials  should  not  be  embedded  in  the 
lETM.  This  will  save  money  and  time  when  either  the  course  or  the  lETM  requires  revision; 
there  will  be  no  need  to  return  to  the  lETM  source  for  curriculum  revisions. 

Billets  should  be  established  for  systems  administration  and  life  cycle  maintenance  of 
hardware,  software,  and  courses.  Existing  instructors  and  support  personnel  cannot  provide  the 
necessary  support  without  additional  training  and  billets.  The  number  of  billets  is  dependent  on 
the  complexity  of  the  automated  classroom.  Using  a  single,  stand-alone  instructor  station  may 
not  require  any  additional  billets. 

Learning  Resource  Centers  should  be  integrated  with  the  classrooms,  be  staffed  by 
instructors  and  be  available  for  after  hours  study.  This  will  free  up  classroom  space  for  other 
courses  and  provide  needed  student  access  for  after  hours  studying. 

Multiple  classrooms  should  be  networked  with  a  central  file  server.  This  will  allow  sharing 
of  data,  easy  reconfiguration  of  classrooms  with  different  courses,  and  make  course  maintenance 
more  efficient.  Life  cycle  maintenance  and  configuration  management  are  more  difficult  if 
classrooms  are  not  networked. 

Instructor  workstations  should  be  available  in  the  instructor  preparation  areas.  This  will 
allow  them  to  personalize  their  materials  and  prepare  for  lectures  without  tying  up  the 
classroom.  These  workstations  should  include  such  peripheral  equipment  as  will  allow 
instructors  to  take  advantage  of  the  new  multimedia  environment. 
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